Using a Longitudinal Student Tracking System to Improve the

This discussion paper considers possible state accountability designs, especially "ValueAdded Models" relying on the California Longitudinal Pupil Achievement Data System (CALPADS). It is intended for a general audience of educational professionals and policy makers, and to inform the deliberations of the PSAA Advisory Committee. Technical challenges are described but are not discussed in depth. Three basic state accountability designs are distinguished and their strengths and weaknesses are briefly described. The paper concludes, as have other studies of the issue, that accountability designs based on tracking individual students' gains have sufficient promise and potential to warrant continued investigation. However, substantial technical hurdles stand in the way of statewide implementation within the next few years, especially for high-stakes accountability purposes in a state as big and complex as California. As with many technical innovations, while potential benefits are real, they may fall well short of some popular claims and expectations. 1 Edward H. Haertel is a Professor in the School of Education, Stanford University. He chairs the Technical Design Group (TDG), which has advised the State of California since 1999 on technical issues in the design and implementation of the API. Dr. Haertel also chairs the API Subcommittee of the Public Schools Accountability Act (PSAA) Advisory Committee. Comments on an earlier draft by the staff of the Policy and Evaluation Division, California Department of Education, are gratefully acknowledged. Any remaining errors or omissions are the responsibility of the author. The opinions expressed are solely those of the author, and no endorsement by the Technical Design Group, the PSAA Advisory Committee, or any other individuals or institutions should be inferred. Longitudinal Tracking and Accountability--2 Using a Longitudinal Student Tracking System to Improve the Design for Public School Accountability in California Edward H. Haertel Stanford University August 2005 Over the years, various accountability designs have been implemented statewide for California's public schools. Details have varied, but all have had one thing in common: None has ever relied on linking individual students' test scores from one year to the next. Thus, while some designs have employed year-to-year gains in test scores at the school level, California has never used a design that required information about the annual test score gains of individual students. An obvious, fundamental requirement for calculating individual students' year-to-year gains is some way to match up each student's currentyear scores with his or her own test scores from one or more previous years--a "longitudinal student tracking system." Senate Bill 257 mandates the implementation of such a statewide student tracking system, known as the California Longitudinal Pupil Achievement Data System (CALPADS). Under CALPADS, Statewide Student Identifiers (SSIDs) will be required on all state assessments beginning in the 2005-06 school year. Thus, it is important to consider how these identifiers might be used to improve the design of California's statewide school accountability system. In particular, considerable interest has been expressed in "value-added" models, incorporating measurements of individual students' growth over time (e.g., Doran & Izumi, 2004). This discussion paper provides an overview of design options for California's statewide school accountability system, with particular attention to designs taking advantage of CALPADS. The alternatives are described in general terms. Many details would have to be specified, and significant technical challenges would arise, in the actual implementation of any such system. Indeed, significant challenges have arisen, and have been met, in the process of implementing and maintaining California's current school accountability system. One message of this paper, however, is that designs that rely on linking individual student records over two or more years would bring much greater challenges. The potential benefits of such models certainly justify further theoretical exploration and perhaps district-level pilot studies, but great caution is urged in moving toward statewide implementation of a high-stakes accountability system using such a model. Three Basic Designs for Accountability Systems Three basic designs can be distinguished for using test data in school accountability systems. In this paper, these will be called Uniform Target (UT), Successive Cohort (SC), and Individual Growth (IG) designs. Distinctions among the three designs are 2 Discussion in this paper is limited to designs for school accountability. "Value-added" models that estimate individual teacher effects are beyond the scope of this paper, and are not addressed. Longitudinal Tracking and Accountability--3 illustrated in Figures 1, 2, and 3 below. Two of these designs are now being used in California. The SC design is used when schools are required to meet annual growth targets established for the Academic Performance Index (API) and the UT design is used when schools are required to meet the State's Annual Measurable Objectives (AMOs) in order to demonstrate Adequate Yearly Progress (AYP). "Value-added models" are all varieties of IG designs. Figure 1 illustrates Uniform Target (UT) designs. To understand this figure, imagine an elementary school with students in grades two through five tested annually. The rows represent test scores from successive years and the columns represent scores from students at successive grades. The ovals show that, for the UT design, data are considered for just one year at a time. Average scores (e.g., reading scores) for each grade (second-grade, third-grade, etc.), or percents of students meeting some standard at each grade (e.g., percents with reading test scores at or above the proficient level) are accumulated over grade levels to arrive at a single number describing the entire school. That number is then compared to some target, which is typically the same (uniform) for all the schools in the state in any given year. The target may rise over time. Thus, schools might be held to a higher performance target in 2008 than in 2007. The UT design might or might not involve combining test scores in different subject areas. The same calculations might or might not also be carried out separately for demographic subgroups within the school. There could be many variations. The defining feature of all 3 Gong (2004) used diagrams like Figures 1, 2, and 3 to explain these designs, although he called them by different names. 4 There may be separate targets for different subject areas; for elementary, middle, and high schools; etc. Figure 1: Uniform Target (UT) Design Year Grade 2 Grade 3 Grade 4 Grade 5